Software Testing – Redefined

By: R. Sankara Narayanan

Part:   1   2  3 

Abstract
Software Testing – the scientific art is being practiced as a separate discipline in all the Software Development Organizations. A number of software testing methodologies have surfaced in recent years to meet the demands of rapid development cycle and increase in number of software products developed. To effectively practice software testing following queries regarding software testing methodologies need to be honestly answered by every tester/manager:

Is it executed, correctly?
Is it correctly executed?


This paper attempts to bring forth the importance of various software testing methodologies. It also facilitates to answer, why, in what scenarios and how should a tester/manager decide upon the various techniques to meet the purpose/end objective of software testing. This paper revolves around functionality testing activities only and it does not take the non-functionality testing activities into consideration.

Definitions/Overview


This paper refers section 3 of Software System Testing Process (herein-after called as SSTP V1.2, http://www.stsociety.org/sstp.htm), written by Software Testing Society (herein-after called as STS), for all the generic terms related to Software Testing Practices. With respect to these definitions, the task of reviewing the software related documents is not considered under the scope of software testing in the remaining of this paper. To facilitate the referring task, few common definitions are made available here:

The term “software testing” is defined as
Evaluating the software (or)
Identifying the condition in the software, of being laid open to something undesirable or injurious.

The term "test case" is defined as a set of instructions that will be followed during the activity of testing of software, and its expected behavior specified against each instruction or collection of instructions.

The term “test library” is defined as the collection of all documents that is created, maintained and used for the purpose of software testing.





Test Methods - Classification


Software Testing can be classified into two major categories based on the tools used.

Scripted Testing – the tools used could be in the form of test cases, checklists, matrices, outlines or any other scripted format. Unscripted Testing – performed without any such tools.

Exploratory Testing (http://www.satisfice.com/articles/what_is_et.htm, refer this page only for the definition of the term and not its explanation) is defined as the activity where the test planning and test execution happens at the same time. The term “test planning” here refers to the approach or way or path that is adopted for the testing. It doesn’t include activities like creating and maintaining test cases, reviewing them and so on. Scripted tools like checklists, matrices or use cases/ requirement documents may aid the Exploratory Testing. At the same time, they may not aid it.

The remaining of this paper talks about how to focus the test efforts across these three methods – Scripted (With well written and maintained Test Library), Scripted Exploratory Testing and Unscripted Exploratory Testing.

Software Testing Process

The current industrial standards/frameworks talks about establishing and maintaining a Software Testing Process, which is institutionalized across the Organization and characterized with the key tasks – Create Software Test Plan and Execute Software Test Plan. The elements of these tasks can be further broken down to

a) Create Software Test Plan


The key task of creating software test plan includes but is not limited to
Learning about the software
Learning about the application of the software
Creating Test Approach
Creating Test Cases
Automating Test Cases
Setting up Test Environment
b) Execute Software Test Plan


The key task of executing software test plan includes but is not limited to
Executing Test Cases
Executing Automated Test Cases
Logging Defects

The sub-tasks of Creating Test Cases and Automating Test Cases are considered very critical and are structured as follows:



Create Test Cases


The task of creating test cases includes but is not limited to
Writing Test Cases
Reviewing Test Cases
Establishing and Maintaining Review Report that includes Review Logs/Defects
Rectifying Test Cases based on Review Report


Automate Test Cases


The task of automating the test cases includes but is not limited to
Writing Automated Test Code
Testing Automated Test Code
Establishing and Maintaining Test Report that includes Test Logs/Defects
Rectifying Automated Test Code based on the Test Report

The above list of identified tasks is generally applied to almost all projects irrelevant of their scale of operation in terms of technology, application, revenue and period.

Why Test Cases?


Test cases help in organizing and managing the task of Software Testing effectively and efficiently. To be precise, they are more helpful and indispensable for the purpose of regression testing and for the mandate of portability and legal requirements to be met. The various objectives of test cases are discussed below.

Objective 1 - Repeatability


Regression testing is defined as the software testing activity with only the following goals:
Check the fix for a software error
Validate that the fix didn’t disturb any other parts of the code

This is a repetitive task and the repeatability of this task is critical. In order to facilitate this, a list of testing activities is performed without compromising the uniformity of the test results. This list of testing activities is generally dictated by the test cases.

Objective 2 - Portability


Typically, a test case is written by a designated person and may be executed by them or any other designated person.

When the test plan demands that one person/group conceptualizes the testing activities and another person/group executes those activities, the portability of that conception is facilitated by writing/sharing the test cases.

There are also occasions where the test cases need to be shipped along with the product in order for the customers to learn about the product.

Objective 3 – Legality


There are few cases where there is a legal requirement for the test cases as such. In order to meet this requirement, test cases shall be created.

If the contractual terms and conditions dictate that one party performs the design/conceptualization of the testing activity and the other party executes them, portability becomes essential thus demanding test cases to ease the situation.

The ‘Accreditation Process’ of government bodies sometimes, dictates that the test cases be written and shipped along with the product.

In order to legally prove that the software test design/conceptualization activity has been performed, the existence of reviewed/approved test cases may get treated as evidence.

Objective 4 - Planning


One is always certain to have a schedule before one starts with the execution of tasks. The schedule features the detailed list of all the tasks to be performed along with their start time and finish time. Organization of testing tasks in the form of test cases generally helps in test planning.

Create Test Cases? - Exploratory Testing?


Section 9 of STS’s SSTP V1.2 states, the test plan shall encompass the activity of creating and maintaining test cases in case of any of the following conditions:

a. Repeatability of the testing activity is essential or critical.
b. Portability of test cases is essential or critical.
c. Test Cases along with Test Results are essential or critical, legally.

Section 10 of STS’s SSTP V1.2 states, the test plan shall not encompass the activity of creating and maintaining test cases and encompass only exploratory testing in case of any of the following conditions:

a. Repeatability of the testing activity is not essential or critical.
b. Portability of test cases is not essential or critical.
c. Test Cases along with Test Results are not essential or critical, legally.

These sections thus make it easier to decide upon the need of the test cases, if we find that the three conditions as mentioned under section 9 are met, test cases are worth to be created and maintained. If not as mentioned under section 10, the test cases are not worth to be created and maintained.

Without Test Cases – What do I miss?


Say, test cases are not chosen to be created. The following questions arise with respect to “Objective – Planning” of Test Cases as mentioned in the section “Why Test Cases?”

How to organize the testing tasks?
How to estimate the testing tasks?
How to plan for the testing tasks?


There is no doubt that test cases are the best candidate to answer all these questions. But they are not the only candidates who can answer and there are many alternatives as discussed in the next section.

Without Test Cases – How to test? – Exploratory Testing


Section 10 of STS’s SSTP V1.2, advocates exploratory testing that falls under the category of Scripted as well as Unscripted Testing. This testing can be applied for all cases that don’t require detailed and well written test cases.

Software Requirement Specification & Software Design Specification


Software Testing is all about verifying and validating the implementation of implicit and explicit customer requirements into the software product. The Software Requirement Specification Document, which carries the detailed list of product features to be built / tested, is an alternative to the test cases to meet the objectives. An explorative tester would be skilled enough to get all the necessary inputs for his/her testing from this document.

Design Specification Document, a more detailed document to the Software Requirement Specification can also be used for the purpose of getting the necessary inputs for testing.

In case the requirement specification information is maintained in requirements management tool an explorative tester, skilled enough to work with these tools would be able to get the required testing inputs.

Checklists


Merriam Webster’s English Dictionary defines Checklists as a list of things to be checked or done. Checklists have short brief description of all the testing tasks – areas/features/functions to be checked and it is not difficult to create them. The checklist creation involves thinking logically to break down the top task of Software Testing. Once the list is made, it can be an input to Exploratory Testing. There are fewer chances to miss out certain tasks if the list is created with good experience. If the confidence level of the test coverage goes down, just update the list

A comprehensive checklist for testing a notepad application would be:


Test for all the menu commands.
Test for append/modification of file.
Test for opening/saving/overwriting the file in the local drive & network share.
Test for printing the document.
Test for multiple instances of notepad in the same desktop.
Review Help documents.

This checklist would make the life easier for an explorative tester as it serves as a road map. Creation and Maintenance of checklists takes lesser time when compared to test cases.

Matrix


Merriam Webster’s English Dictionary defines, Matrix as something resembling a mathematical matrix especially in rectangular arrangement of elements into rows and columns. Correspondingly, if there are similar kinds of actions to be performed with different instances/ situations, a matrix can be easily prepared offering a user-friendly tool to the tester.

For instance, refer Table 1. This matrix is prepared for a website to validate the login page, registration page and their text GUI elements. This matrix contains the list of Page + Field details across the rows (row headers) and the list of actions to be performed with text elements, across the columns (column headers). The cells at the row-column intersections can have values NA (Not Applicable), D (Desired) and ND (Not Desired).

If a cell has NA that means the action that is specified in the column heading shall not get executed for the Page + Field that is mentioned in the row header.

If a cell has D that means the action that is specified in the column heading shall get executed for the Page + Field that is mentioned in the row header. That action is a desirable action and the explorative tester would check for the application to perform that action.

If a cell has ND that means the action that is specified in the column heading shall get executed for the Page + Field that is mentioned in the row header. That action is a non-desirable action and the explorative tester would check for the application not to perform that action.

(Continued on next part...)

Part:   1   2  3 

Other Resource

... to read more articles, visit http://sqa.fyicenter.com/art/